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of distance collaborative processes, including telephone, e-mail, and 
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Expert Review at a Distance: A Hybrid Approach 



The purpose of this paper is to describe a distance collaborative evaluation method 
employed to analyze an asynchronous web-based course. This course is offered through the 
Instructional Technology department at Virginia Tech as part of a distance education Master's 
program. An initial Expert Review of the course was commissioned by the Virginia Tech 
Instructional Technology department as part of an assignment for a graduate course in product 
evaluation. A first year doctoral student from Virginia Tech followed specific guidelines 
provided by the department to complete the review process. Several leading instructional design 
models were utilized to conduct this initial review and will be described below. After completion 
of this initial review, a second doctoral student, this one from the University of Georgia, 
reviewed both the web-based course and the first reviewer's comments. The cumulative review 
of the first two students was then passed on to the course developer. The course developer added 
his own comments and incorporated student feedback into this collaborative distance-based 
review. All of the information was then re-circulated and a consensus reached that incorporated 
all the information into a collective list of evaluative recommendations. All correspondence 
between the three reviewers was accomplished using a variety of distance collaborative 
processes, including phone, email and ATM VTEL videoconferencing. Research concerning the 
transactional distance that can occur in a distance environment was substantiated (Moore, 1996; 
Wolcott, 1996). The evaluated course assisted Virginia teachers in creating their own educational 
web sites. 



Web Site Development: Instructional Technology Master’s Module 4 

With the recent adoption and mandate of the Virginia Standards of Learning (SOLs), k-12 
instructors throughout the state are now required to take coursework for re-certification that 
demonstrates a set of core technology competencies. In efforts to support this statewide 
endeavor, Virginia Tech developed and implemented an Instructional Technology Distance 
Education Masters Program (ITMA). The instructional delivery methods Virginia Tech utilizes 
employ an assortment of strategies such as intensive workshop-style courses offered within 
individual school districts (on-site); interactive television courses offered through a number of 
state-wide facilities; web-based courses; week-long, on-campus summer courses; open-studio 
development; and individual computer-based programs. 

This online program consists of 13 unique ITMA modules that can be taken independently 
of each other as needed to satisfy Virginia SOL re-certification requirements, or in total toward 
the Virginia Tech Master's degree program. 

Our evaluation concerned itself specifically with "Module 4: Web Site Development" 
( http://www.itrna.vt.edU/itmal/iriodules/webdev/~) . The intent of this module is to provide 
learners with basic HTML design skills using Netscape Composer, such that they develop a 7-8 
page standalone web site. Should the learners not be taking this module independently, but as 
part of the overall Virginia Tech ITMA program, their product will be integrated into a larger 
electronic portfolio. There were 51 students who participated in the first iteration of the module. 
These students were divided into three cohort groups based on their geographic location within 
the state of Virginia. They constituted a diverse mixture of elementary and middle 
schoolteachers, librarians, and instructional technologists in the public school system. Our expert 
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review was performed to increase the effectiveness of the instruction for this diverse group of 
learners. 



Expert Review: The Models and Theories Employed 

The overall model of the Expert Review was patterned after the procedure outlined in the 
book Planning and Conducting Formative Evaluations (Tessmer, 1998). Tessmer delineates the 
steps and values behind conducting the following types of reviews: One-to-One, Expert Review, 
Small Group Evaluation, and Field-testing. The Expert Review conducted on Module 4 focused 
specifically on the following areas: General product description, instructional components, 
instructional contexts and functions, and overall instructional message display. A brief 
description of these areas is warranted. 

The general product description delineated the developers, clients, module objectives, 
learning environment and media characteristics incorporated in Module 4. The "instructional 
components" section evaluated the overall structure of the module from a Walt Dick and Lou 
Carey instructional design perspective (1996). In addition, several other ID research theories 
were employed which will be discussed later. The instructional context of the module was 
examined to evaluate the synergism between all the factors influencing the appropriateness of 
context chosen and under what functions it was intended to facilitate instruction. The overall 
message display was additionally evaluated with respect to the following: layout, color, density 
of information, perceived units of information, rate of information presentation, and 
appropriateness of images and/or audio used. The evaluators applied several ID research 
modules in evaluating the components described above. 

To evaluate the overall module from a "Big Picture" perspective the Walt Dick and Lou 
Carey model was used (Dick & Carey, 1996). The following components were examined in the 
module for their inclusion of appropriate sub-component characteristics: Introduction, activity, 
practice & feedback, review, assessment and transfer. Within this overall structure, the individual 
instructional objectives were examined using the methodology cited in Sullivan & Higgins 1993 
book: Teaching for Competence (Sullivan & Higgins, 1983). Thus, instructional objectives were 
examined with respect to observable and measured behaviors, with proper attention to 
conditional givens and standards. 

The work of Tessmer & Richey (1997) was used to classify and evaluate the module with 
respect to the appropriateness of context and its function within the overall design stages of 
orienting, instructional, or transfer (Tessmer & Richey, 1997). Module 4 employed the contexts 
of "Real" and "Tutorial" in an appropriate fashion. 

J.M. Keller's ARCS model of instructional design was utilized to evaluate the module from 
a motivational perspective, attempting to identify if the following strategies were implored: 
Attention, Relevance, Confidence, and Satisfaction (Keller, 1987). Please see Appendix 1 for a 
review of Keller’s ARCS model. 

Finally, the work of Gagne and Driscoll was utilized to measure the appropriateness of 
instructional objectives to specific tasks or lessons at hand (Gagne & Driscoll, 1998). Please see 
Appendix 2 for a template of the instructional design model and theories applied to evaluating 
module 4. Appendix 3 presents a section of one of the expert reviewers evaluative comments. 
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What was interesting is the nature in which this expert review was conducted in a distance 
collaborative nature. 



Expert Review at a Distance: The Process 

The original Expert Review (ER) paper was circulated to the second distant expert 
reviewer. The second reviewer then visited the Module 04 web site, reviewed the ER paper, and 
evaluated the initial reviewer's criteria against the originally commissioned guidelines and the 
actual module, inserting comments into the original ER paper using a different colored font. The 
updated ER paper with comments was then circulated to the module developer and the initial 
reviewer. 

Next the developer's implementation perspectives were incorporated as a separate section 
at the bottom of this evaluation paper, describing the technical difficulties encountered during the 
module's first field test, including student impressions of Module 4's effectiveness. This updated 
document was then sent to both reviewers who additionally commented on the developer's 
implementation analysis. Finally, having now seen both sets of Expert Review comments in a 
single document, the developer offered his impression of the expert reviews. Needless to say, 
this asynchronous distance evaluation approach involved several iterations of circular review, 
providing a thorough analysis of the module from an ID Expert Review perspective. 

The techniques employed in the collaboration process utilized several forms of distance 
communication. The review document was created in Microsoft Word and passed between 
reviewers via email attachment. Discussion and debate between the reviewers took place using 
email, teleconferencing, and videoconferencing. A significant portion of the review occurred via 
asynchronous email exchanges allowing the experts to ponder their critiques prior to submission 
at a pace and place of choice. Several teleconferences occurred at the key junctures in the review 
process when decisions needed to be made in concert with all three members with prompt 
asynchronous feedback. ATM VTEL videoconferencing provided the not only prompt 
synchronous feedback, but the most realism in conveying the strength of a reviewer’s conviction 
about a recommendation or revision. Unfortunately, the expense of conducting VTEL 
communication limited this type of exchange to only brief communique. 

The important realization to come out of this process is that without in-person, face-to-face 
contact there is a higher probability for miscommunication to occur between participants, 
especially when those participants have never met face-to-face. At several points during this 
collaborative project there was confusion due to misunderstandings created by the psychological 
distance between the three reviewers. This type of transactional or psychological distance has 
been documented in previous distance education literature (Moore 1996; Wolcott, 1996) and 
communicative efforts needed to be intentionally tempered via this distance-based critique. The 
synopsis of the aggregated conclusions between the two reviewers and the module 4 developer 
were quite interesting. 



Expert Review: The Recommendations 

Both expert reviewers agreed that overall the site was exemplary. The module was well 
designed, easy to understand, implement and execute. The site was additionally aesthetically 
pleasing from a graphical design standpoint. Most importantly the asynchronous module was 
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successful in its instructional goal of facilitating the learner with the creation of standalone web 
sites. Both the director of the ITMA program, Dr. Greg Sherman, and the participants 
commented favorably in survey reviews at the completion of the module. 

From an instructional design standpoint, only a few suggestions were put forth, which may 
improve the effectiveness and efficiency of the module. The comments that follow were the 
collective agreement from all three parties. First, there may be a benefit to incorporating an 
official assessment for each sub lesson within the overall unit by way of a rubric. This rubric 
would specify evaluation criteria, thus providing a guidepost to the learners prior to their project 
development, and a way for learners metacognitively self assess their own work prior to 
submission for grading. In addition, the Composer tutorial should include instructions specific to 
the Macintosh platform. At the time the module was initially offered for course credit, there were 
a significant number of students using the Macintosh platform. By offering additional 
instructions or screen snapshots unique to the Netscape Macintosh Composer platform, the 
module might alleviate any confusion with platform specific issues. This could be done by 
incorporating Macintosh examples in the existing tutorial, or by creating a new tutorial specific 
to the platform. 

Also, a separate lesson dealing specifically with tables may be warranted, incorporating 
additional instruction and a student-to-student collaborative constructivist model. The module 
developer share with the expert evaluators that the distance teachers taking module 4 frequently 
wanted to create more controlled html page layouts and tables are the way to do this within 
Netscape Composer. Possibly by structuring in a peer-support or mentor-based system, the 
module could engage a collaborative dialog regarding the creation of certain “table” page layout 
looks. 

Finally, the overall module may be made more effective should all lessons utilize more 
examples of student work tied to reflective student-student collaboration. Interaction that fosters 
peer collaboration, reflection, and assessment has been shown to enhance the distance learning 
process. It should be made clear though that these suggestions are unproven at this juncture, and 
their success must still be bom out in future revisions through evaluative field-testing. 



Closing 

This paper describes a distance collaborative evaluation method employed to analyze an 
asynchronous web-based course in web design for teachers seeking recertification in the state of 
Virginia. The course is offered through the Instructional Technology department at Virginia 
Tech as part of a distance education Master's program. Various asynchronous and synchronous 
technologies were employed to conduct the expert review of the web-based course including 
Internet email with attachments, teleconferences and ATM VTEL videoconferencing. Issues 
concerning psychological or transactional distance were addressed throughout these various 
modes of communication as the expert reviewers communicated with the course developer. 
Several models of instructional design were utilized for the ID review including that of Dick and 
Carey (1996), Tessmer and Richey (1997), J.M Keller (1987) and Gagne and Driscoll (1998). 
The area of instructional context was also evaluated and a concise model has been provided in 
the appendix of this paper. 
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The process of conducting an expert review at a distance is a viable alternative when the 
experts are geographically dispersed and unable to meet with the developer for a face-to-face 
exchange. It was found that communique between the evaluators and the designer needs to 
structured in such a way to overcome the barriers erected by temporal and physical separation, 
but valuable and worthwhile evaluation may be achieved using a multi-modal distance-based 
approach. 
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Appendix 1 : 



Keller's ARCS Model of Addressing Motivation in the ID Process 





Attention Strategies 


Incongruity, 

Conflict 


Introduce a fact that seems to contradict the learner’s past experience. 

Present an example that does not seem to exemplify a given concept. 

Introduce two equally plausible facts or principles, only one of which can be true. 
Play devil’s advocate. 


Concreteness 


Show visual representations of any important object or set of ideas or relationships. 
Give examples of every instructionally important concept or principle. 

Use content-related anecdotes, case studies, biographies, etc. 


Variability 


In stand up delivery, vary the tone of your voice, and use body movement, pauses, and 
props. 

Vary the format of instruction according to the attention span of the audience. 

Vary the medium of instruction. 

Break up print materials or (displays) by use of white space, visuals, tables, different 
typefaces, etc. 

Change the style of presentation. 

Shift between student-instructor interaction and student-student interface. 


Humor 


Where appropriate, use plays on words during redundant information presentation. 
Use humorous introductions. 

Use humorous analogies to explain and summarize. 


Inquiry 


Use creativity techniques to have learners create unusual analogies and associations to 
the content. 

Build in problem solving activities at regular intervals. 

Give learners the opportunity to select topics, projects and assignments that appeal to 
their curiosity and need to explore. 


Participation 


Use games, role-play, or simulations that require learner participation. 

Relevance Strategies 


Experience 


State explicitly how the instruction builds on the learner’s existing skills. 
Use analogies familiar to the learner from past experience. 

Find out what the learner’s interests are and relate them to the instruction. 


Present Worth 


State explicitly the present intrinsic value of learning the content, as distinct from its 
value as a link to future goals. 


Future Usefulness 


State explicitly how the instruction relates to future activities of the learner. 
Ask learners to relate the instruction to their own future goals (future wheel). 


Need Matching 


To enhance achievement striving behavior, provide opportunities to achieve standards 
of excellence under conditions of moderate risk. 

To make instruction responsive to the power motive, provide opportunities for 
responsibility, authority, and interpersonal influence. 

To satisfy the need for affiliation, establish trust and provide opportunities for no-risk, 
cooperative interaction. 


Modeling 


Bring in alumni of the course as enthusiastic guest lecturers. 

In a self-paced course, use those who finish first as deputy tutors. 
Model enthusiasm for the subject taught. 


Choice 


Provide meaningful alternative methods for accomplishing a goal. 
Provide personal choices for organizing one’s work. 
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Confidence Strategies 


Learning 

Requirements 


Incorporate clearly stated, appealing learning goals into instructional materials. 


Provide self-evaluation tools which are based on clearly stated goals. 


Explain the criteria for evaluation of performance. 


Difficulty 


Organize materials on an increasing level of difficulty; that is, structure the learning 
material to provide a “conquerable” challenge. 


Expectations 


Include statements about the likelihood of success with given amounts of effort and 
ability. 


Teach students how to develop a plan of work that will result in goal accomplishment. 


Help students set realistic goals. 


Attributions 


Attribute student success to effort rather than luck or ease of task when appropriate (i.e. 
when you know it’s true!). 


Encourage student efforts to verbalize appropriate attributions for both success and 
failures. 


Self-Confidence 


Allow students opportunity to become increasingly independent in learning and 
practicing a skill. 


Have students learn new skills under low risk conditions, but practice performance of 
well-learned tasks under realistic conditions. 


Help students understand that the pursuit of excellence does not mean that anything 
short of perfection is failure; learn to feel good about genuine accomplishment. 


Satisfaction strategies 


Natural 

Consequences 


Allow a student to use a newly acquired skill in a realistic setting as soon as possible. 


Verbally reinforce a student’s intrinsic pride in accomplishing a difficult task. 


Allow a student who masters a task to help others who have not yet done so. 


Unexpected 

Rewards 


Reward intrinsically interesting task performance with unexpected, non-contingent 
rewards. 


Reward boring tasks with extrinsic, anticipated rewards. 


Positive Outcomes 


Give verbal praise for successful progress of accomplishment. 


Give personal attention to students. 


Provide informative, helpful feedback when it is immediately useful. 


Provide motivating feedback (praise) immediately following task performance. 


Negative Outcomes 


Avoid the use of threats as a means of obtaining task performance. 


Avoid surveillance (as opposed to positive attention). 


Avoid external performance evaluations whenever it is possible to help the student 
evaluate his or her own work. 


Scheduling 


Provide frequent reinforcements when a student is learning a new task. 


Provide intermittent reinforcement as a student becomes more competent at a task. 


Vary the schedule of reinforcements in terms of both interval and quantity. 



Keller, J. M. (1987). Development and use of the ARCS model of instructional design. Journal 
of Instructional Development. 10 (3), 2-10. 
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Appendix 2: 



Instructional Design Template use for Evaluation 
Basic Instructional Design Model 

Once a meaningful, purposeful context for a particular instructional goal is identified and 
described, provisions must be made for addressing the important elements of each ID 
component. These elements may be addressed before, during, or after the learners “experience” 
the activities associated with the context. 



Component 


Elements 


Context 

Function 7 


Introduction 


• Gain learner attention 

• Articulate in some way the SKA 1 already needed to succeed 
within the new learning environment 

• Identify opportunities in which learners will relate what is 
about to be learned (goal) to what they already know how to 
do 

• Inform learners of objectives (o.k. to be vague) 

• Fit objective(s) into a "Big Picture" 2 

• Present the utility (relevance) of the SKA to be learned 

• Clearly identify the incentives/rewards for learning the SKA 

and succeeding within the learning environment 

• Establish clearly-perceived learner accountability, role(s) and 
task(s) within the learning environment 

• Establish clearly-perceived instructor role(s) and learner 
support mechanisms 

• Employ specific content-area methodologies if applicable 
(ensuring that all other elements are properly addressed) 3 


Orienting 


13 

c 

o 




Activities 


• Establish in context the appropriate conditions for the typetsl 

of new SKA facilitated 4 

• Relate all SKA to a clearly-defined content domain 5 

• Provide learner guidance for learners as they apply 

information presented in context to SKA being facilitated 

• Present a variety of clear, concrete examples and 

nonexamples in context 

• Provide opportunities for learners to explore the learning 
environment with minimal instructor guidance and 
intervention 
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